An Evaluation of the Accuracy of Capturing User Intent for Information Retrieval

This paper reports our evaluation of the accuracy of capturing a user’s intent in an informationseeking task. Specifically, we would like to assess how accurately a user’s short-term goals, methods, and context in an information seeking task have been captured. Our method is to compare a machine-generated model against a human-generated model with 5 users using the CACM collection. Our results demonstrate a good coverage of human-generated user models by machinegenerated models in terms of short-term goals, methods and context. The similarity between a real user and our approach in terms of context agrees with the similarities of human -generated ontologies using the same metrics in existing studies from the ontology community. Furthermore, our results show that the similarities between a human and machine in terms of short term goals and methods, which affect the relevancy assessment process, agree with the overlap among people while assessing relevancy documents in the existing study from the information retrieval community.